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Bldprinting Teacher ticerislng Tests.:- Developing 
Domain Specifications frpni -Gob Analysts Results 



An increasing number of states are requiring^>^feSatbVs to take 
certification tests in order vto obtain a license t^.leapfe SWhile this 



trend shows little sign of waning^ there fs^little^t^ gui-i^'^th^^ . v 
developer of teacher licensing tests in defining test speciffcatiom in 
a legally and techhTcally sound mannerly This, the ligal 

and technical' issues that must be 'considered in developing 
specifications for teacher licensing Jests, A practical^ technically 
sound, and legally def^n^sfe^^^^^ developing specif ic^^^^ 

fbr teacher licencing tests. frOT analysis; results is, presented. 



Bluprifitlflg Teacher Licensing Tests: Developiag 
Domain Specifications, fro^^ dob Analysis Results 



i 9ver the past two decades ah increasing nuriiber of stares have required' 
educatdrs to pass one or more teits fh order to obtain i license to teach. 



Theres^are currently 30 stages that have mandated some fonn of competency 
testing for teachers and at least ;i3^hers are seriously dons ideFing^^^^^^ 
in this direction. While there are numerous tools available to test 
developers for identifying the doma«in of knowledge to be - included on tests of 
student competency, there i^, little to guide the developer in specifying the 
^ domain of content to be included in licensing tests for teachers. This 'paper 
: discusses the use of 30b analysis results in specifying the domain of content 
\ to be included in, licensing tests for teachers ^ . The legal and technical 
issues that must be considered in developing test Specif ica^idns from job 
analysis results arg addressed and the specific methodology employed for . 
I develojDing specificatibns in a recent teacher licensing test (^eldpment 
^^fP^I described. An overview of job analysis is-prov^ be^ost>t^rdv 
a context for the. discussion of , these issues. 



^Job An a lysi s 



instruinent' designed for certification or •licerisirigV asM*s the case in 
teacher certification t^^^ must be shown to be job relateldr' It mu^^^^^ 



fairly measure the cdntent knowledge relevant to the job as performed by~ 
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tires^nt job irieumberits. Ditermlning the job relatedneSs df iontent s^i.ected 
for inclusion; in certification tests is both endorsed in the APA Principles 
for the Validation and Use of P^s ohne-I Selecti on ProGedufes (19891 ani ,/ . 

f h'e gui del'i ries requ ire that thi cr i tef i a tisid as a "lasi s. of certif i ca^i on uftist 
bear ah enpiricarK and Tdgical relatiorii5hip |o successful job perfdnnanc 
purposes of teacher certification, this suggests that test content should 
reflect the Gojiteht knowledge or pedagogical skills required for teptchihg.i^ 
While there ^e a number of ways in which this; domai;n of knowledge 'can be}!' 
identified, a systematic 36b analysis ts r^cbinrended to establish an empirical 
and Idgihal relationship to teacher perfoma^ . : ^ > v^v 

Job Analysis Approaches . Job analysis is a process of systematically 
collecting information about the e?feniehts,of 'a job. While job ahalysis has \ 
been rdutinily used in personnel -related are as -for close to a century, it is, 
only within the past few decades that it has been .enployed in personnel 
testing . • .. . " ^ ; ■ v' "J ■ i , ■ ; ^ 

A variety of apprdaehes to assessing the\element_s of a given, work' 
situation are available; hdwe^eri regardlerf^of the selected method, most 
approaches include some detemiinati_dn of the critical • and frequihtly per - 
formed' elements df the job. ^ Importance (critical ity or essentiality) and 
frequehcy of perfdrmahce (tirnl^t^nt or percentage of timi con|Umed on job) 
are the two key dimensions underlying most job analysis approaches. Within 
the teacher eeftif ication arena, thjs would generally take the form of 
assessing the important and frequently applied teaching skills or cpntent 



knowledge in the instructional setting^ 



Wl- 



r'H 



\ Job ahaiysis apprfdachf s can be seen to varjr' a'Vong '4 number; of difflefi-^ 
f i'/ si(^&L : bevi^ie, Asfeflal^v arid Sistrtirifc (i98ii have dflineaiied' thr^^ K^^ J 



nmensipns along which job analyses vfiry: 




type of descnptor or element Used to descnbe the 30b/ ^ 
tfie.source of job information, and 

data collection methadology. f ! ■ ' 

the descriptors, used to describe a |joK are tasks, acttvities, 
/ /ikill I; knowledge, and perso.nal characteristics^ A number of sources of job 
^ informatiijn are potehti ally available these include j'db ihcuitlbentSi 
. saprvisors, trained job analysts^ and writteiii docurherits., i Data col lectioir 
mfethods "include questionnaires, interviews^ .observation, diaries, and actual 
job* performance. .AlthbUgh; it is clear that many approaches to conducting Job * 
analyses are available the application of job' analysis methbdblbg^^ 

certtf ication testing has been somewhat limited. 

■ ' ^ ' " . ■ ■' - ' ' 

. ' - ■ . ' - ■ -■- , - • . _ - J 

Gurrent applications- bf_ job analysis methodology to teacher ceftif i- 

catTorTtesting fs presented beloWi Other Job analysis approaches derived from %. 
the three dimensions cited previously, with potential applications to teacher -v 

: .7 - ' - . : - - ■. ■-, : : . 

certification, are offered following the discussion of current applications* 



Job Analysis Applications , Gob analysis' has been used in the conte^t^ 
validation of teacher certification tests in a number of states. 'Among the 
states that have conducted job analyses as part of their teacher certification 
test /devel opment efforts are (SeorgiaT Alabama, Oklahoma, South SaroHna, -and 
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^West V^rgiriiai in all five cases a survey approaph was ^tt^^^ sample pf 



: ; educKors within .thje ^tat§ were sent a survey instrument requesting -than ta- , 
rate on a Likert-type scale a -series of contgnt objectives^ developed by ^ 
panels of content experts, in terms of t"he amount of time spent teaching or " 

; using* the objectives and the extent to which the objectives were essiential to 
the field. Bassoon the job^ analysis res^ultSj thos^ ob^'ectives; found to^be 
most 30b related were included m the cdhterit of the exarfiiriatiolis. In some - 

pase'l ah interview pl'ocedure was used with a sample of educators to supplement 

.._ _ •■ - y _ . ■ ' _ . • - __ 

the quantitative ratings and father further information about 'job content; . 



Similar* procedures wers used in the d^velppm|nt &f the Florida Teacher 

_: ._ _ _ ______ . ■' . ; _ _^ _.. '^IJ. ■ 1 .* .4 

Certification Examination, Teacher competencies (objectives) were developed ^ 

by a panel of teacher educators, the cdrnpetencies were then sent to a sample 

of e^ducators who rated the competencies in. terrns^^^W their perceived 

"impbrtafice-'vto, the fields No ratings of "frequency of use" -or "time spent 

using" were collected. / * . ^ ' . y '- '^^ 



i Similar" procedures *ave been used for more process-or.iented assessment 
measures developed for use iti teacher; certification. ' The Bs^ Professional 
^Studies ExamiriaJ:idri developed 'in Alabama, to assess feriowledge of pe^ 
sk^l Is relied ort job arralysii; for detemihihg the cbnt^rit to appear on the, 
test, ft sample of educators across teaching fields r^'ted the frequency witn 
which -pedagogical ski^ used "and the importance of those skills.; The* 

content of the Perfon^^^ in South Cardliha 

waS; defined th?*ou§h a jb& Shalysii. Again,-^sing ^ sUryey approach, a sample 
of South Carol ina educators rated the -impdrtance and: frequency of use (as wel 1 



Tfie recently revisecl "Ebnfnbn"; portiph of the NTE ^did not rejy. on fob . - 
analysis data in defining the conte^^^ included on the exaniiriatibrii 

Heres state representatfvSs .were, surveyed to deterrTiine tfie exten^^^^ 

^P#t4ag e g4eal4 y r e l at ^d^^sTjHi^ ^are irtipbrtant fdr purposes of ^ 
teacher certif icatidh:. ^ > / 

i , • ■" ■ '■■ ' - ■ ■ ■ , ■ ■ '• . ■ . • . 

- job' Analysis Alteroatives^ . While job analyses conducted for current teacher 

fijcertification tests .h^ : 

.questionnaires .requesting 30b incumbentS; to rate\proposed test content in 

terrns of importance and frequency of use, other alternatives suitable for use 

' - . -. * - ■ fc. . . . - ■ 

in teacher certification testing are availaBle. RecoiSnendations^ for job 

analysis alternatives based on (1) type of 'data collection methodology and (2) 
source of Job infonnatibri are prbvidea belswe .' • 

- ' fffacher certification test developnjent efforts, to date, hav6 relied on ; 
" the CO 1 lection of job iriformatibri from" a crbss secti bn of job incumbents 
.reflecting^ the teaching area for which the -measure is b6ing developed. 

Atterhativis toi-thi U|e of a. cross section of job ihcumbe/i^s ihcjude t^e - 
cbllectibh of jbb infb-ritiatibn from supervisors or solel'y from superior 
perfbrmers bh the job. Previous research comparing the job irifbrmatioh; 
obtained' from -job incumbents and other observers is conflicting (tevih^ et 

' : : - . , _ ; v.. 

al.,' 1981). While the infbrmation obtained from incumbents and other 
Observers appears ta be consistent in some job settings, Levine et al. ^ (1981) 
suggest that In other settings ihcumbehts tend to provide less acdurate *'^; 



accounts d job cbntent. have been made tb- 



invelti§a$e the o'nfo'rniatieTf ^ from teachers as cdmparea to th6 



Jftformation obtained from other observers- in the instructional environmelit, 
anovthe acfuracy^of teacher/educator supplied information remains to be 
ex-pldredr 



SjmllarlVi little effort has been made to compare the -job information . 
obtained 'froin\ teachers judged as Superior to educators judged to be poor 
performers. ^I^fiile Levine et al; (1991); In their recent i discus^jofl- of 'job 
analysis methodology j suggest tH§t/there are few differences in J:He job 
information obtained from superior and less capable performers in a^varie'ty bf 
job. set tingsf this r^ains t^^he/^efi^red in the instructidnal;i setting; Tfie 
valiidity of tfeiacliercertifi cation tests based on job .content defined solely by 
Superior per;formers coul^ :be brought into question as these measures are • ' 



generally designer 



■as minimum' competency as'sessments; 



> There are a number of S^ternati ve data col lection methods that feould be 
applied ito analysts /efforts- undertaken, fq^ teacher certificato>n:vtestv:Vr ■ ^ 
development purposes. Among the alternati-ve^ to the surv^: ^uestidnnaire v 



approach (which has! been -^he primary" detta^ collection method empTbMd^jfor 



A 



teacher certTflcatioh toting tO'^dat^) are-fl); obseWation, (2) ; critical 



Incident technique vtlFUntfgani i^54)v''(3)v5loqtim^^^ (4) ^^rotip 



discussipn. 



'1 , 



I ^ Job analysis data ctilleetibil using >fes^^ relies: on trained 

oBier^/ers observing, the: p^^ df^tjol^ i^cu^^ the re^alm of ■ 

teacher certific^ioh, this would :Th\^^ the ; 

cLassroorff behav^^T^ other instructional; ;pe^^ to ascertain 

the contehi:; of ' the J While providing a the' dob ; ; 

'cdntifit;^ -tfi^^^ approach is \qdest^haBTe becad its / . 

obtrusi'Venfiss and resources required. TKis is particiularly true in the case' 
df content Jcndw^ exaitiihatidns developed for teacher certif ication 
purposes; repeated observat time would be 

re^ijired»t%provide an accurate assessment of the content knowlidge required , 
on 'the dc 



The dritfcal incident approach i deve^^ by Flanagan (1954)j involves the 
identificatidri of job events - that have resulted in either inferior, or superTdr 
/ -peH^dnTiance (i.e.^ that elicit behaviors necessary for succ^^ 

performance). ■ A; large number 'of incidents^ are cgllected from Jo6 inciimbenls ; 
■ (through diaries, interviews, etc.) arid are used to' determirii 
- a^ necessary to be effective oh tfte jbbv In applicatidri; to 1^^ 
cirtifieatidn testing, this would require the ^elidititioR^ 6^^^ 
; V incidents in the instructional setting from a pool of instruction 

personnel, this approach is potentially useful for' the development 'of teacher 

on pedagogical skills; however, ' the ^ 
criticaU incident technique/appears to. have little application to contsnt 
^. knowledge-oriented measures. Levine? et'al., tl981). report that Jbhl 5 approach / 
, > ' was not favored by. experieneed'"jo.b anal-ysts for use -in persoane'l ^election.- 
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The final two data cd 11 ^ctibh Approaches with potential' appli^tibri to : • 
teacher certification document review and group discussidni Bbctimerit , 

s-^ tie use of ava 4^-a fe^ e lit ^atw^ ^f^fn^ for 
determining necessary job content i flere^ job descriptions ari^d ath 
documentation would be reviewed to determine the critical aspects 'df t)ie job 
:to include in' a personnel selection instrument,— To the extent t^ 
documents exist within educational ehvirohmeht^ approach; could be 

employed. In fact^ the review of 'sjiicji documents is already carrietf out, to. a 
limited extent^ in the definition Vdf cdhten^ knowledge or skills to be ' 
Included bri; job aria^ used iif existing teacher ^ 

ijcertil^cation ^est development projects. Similarly, t^^^ > 
jiiethdd^.to. be considered--group discussibrr--has beefr u the developffient of ^ 

Existing teach e»j cert tests. In the de^ldpment of 

tests f^ 

of expgrt^ 'w^re fcpn^^ in the respective content areas to generate cdnteh^^ 
ybr iiTtlusian^d^^ This could be expanded^ 

i;ta include supervisors "and. incumbent^ in the respective areas-who wduTa , : , 



•Ml 



formally rate the ic^bwledges and skil Is idintif ied in terms of .their 
importance. 



While there iS: 



Tttle questibh that teather Vicehsihg tests should be. Job 



analysis based, there is little to guide the developer of teacher 1 tcensing 



. i->^*-:■ 



t,es,ts in making thl leap from job analysis to domain specifications. 
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. Domain 
Specifications 



it 



. Thl^^' pr^bblenfS^ in a recent teitVd^el^^ eff ort . 1 

designed to prddtifi|f li^^^^ of Business education;. 




anatysi s : resul ts vane de^i bed 5el ow . 



acmTairt Specif ic^^ Procedure '-'Mv.rx 



-.-1 • '"'j^Jb ' 



-I Domain Def1ni>t ion . ^The firft steja in specifying t^^ 

of content to 6e included on the licerising test for Business Education 
:teachers was to del 1 rieiite the job content was def tried as^ i sft df ^ 
cbgniti ve objectives defining the content required arid the (taxonomic) 
l^yei^at it needs to be known; A comprehensive set of ^^ d^^ 

can be'derived through a viV'i^ty df methods iOcludihgr. Review. i9f existing 
job dif inil^dris, certificatidn standards and curriculum maieriiils, expert 
panels^ dbservatibh, and interviews. ' , ^ 

Fdr the develdprnent effdrt^ a cdmprehensive set of cognitive 
objectives was defined based dri a review of teacher certification y : ^ f 
standards, curriculum materials and tel^tboblcs. - The se'^df i approxijnatel^^^ 
200 objectives were cateiorized; into five. major content subareas: 
Bu'sihess Conmuni'catton'SklTTs',^^ Math. and Accouritingj General ;> ■ ^ • 

Business, Business'' and Ybutft Organizations; and Information Processing. ^ « 



The set ^ of' ob ject i ves^ devel oped was ' rev i ewed • by a panel of eight 

../!■.,,■ : T"-^- '- 

- ■. ' ' ^ ^ ■ ' . ' ^ * ' • * ■ ■ ■■ ■ : ' 



• practicing Business Educators were revised^ added^ and^ : 

deleted as necessary to ensure tha^^ ^t reflected the f teW of Business 
Education as^.it is taugfit and understood in the jurisdiction in which th^ ' 
vtest^wa^ to be used for ^ice^stng.v 
. ; Conduct - job Ah a l ys i s > Deteyinining the 16b-te1 atedness of objectives 
selected for uie in certification testing is a requirement of the Equal ^ 
Empldyment Opportunity Guideline^ Acfeordirig to the guidelines, the 
criteri a used: as a basis of; certification must^ bear an empirical and , 
iQgidal r6lationsf)ip to t|e actual ^ob content. : The job |inalysis • 
procedure described below Rfovides tHMi info^^r^^^ 

db jecti ves wh i eh are nids t represent at i y| of th e cbntent knbWt edge requ i r ed 
for teacfiing •Business Edu^^ 

At the concltision of the objective review,v the Business .Education 
fielB ftad ieen defined by a set of oBj^ctives approved by the* panel of * ^ 
experts, pese objectives were then submUt^ job analysis to / . 

determine their "jdb-rel atedness.^ content experts was asked 

to indicate, for each bbjectivei the ambuht/bf time spent teachitig'or ; ; 
usin^ the bbjective and the extent to which the objective was essential to 
the field.. Obtaining ratings of "time spent" and essential ity" fr0r|^ 
panels: of cbntent experts' (i as been use^ number bf tasR and jbb 
an^ys^ servih§ as -a basis for the development of certification and 
licensing iests; - ■ * \ £ 



10 



-. s . Time- spent teaching or usiiii^an- b was rated on a five-poirit 
scale ranging from "very litt^^ 

: . waS; atso scale ranging from ''essential to a very 

/: little extent" to "essential to k very great extent." . . 

, : Anal^^ Job Analysis Results ^ The objective ratings were analyzed to 
A. " the extent to which each objective was "job-related." ■ 

J^^^ O Cdniputed for eich "object tv^ on the two rating ^ 

;, ^ '^^scaies completed^y the jia'nfel of BustlVe?.^-:'l'clBcai:l6h •'tont'Sni''jix'pirt$.v - Th4'."* 
i ' • ' ' : average (liiean^; amountjof time sp^nt. teaching or ijsing; tfie^ o^jeptiye^^^^^^ 

the average (mean) extent to which educatol^s fett the objective was, 
s _:^-issential to the field were detemined. Final lyi the grand mean rating 

' I and standard deviation across all objectives rated was deterniihed. Based 
.. on tHe grand mean and\grand standard deviation the objectives were divided 
into four categories for each rating scale: 



-i Preferred- M Objectives with job analysis ratings more than one 
Standard deviation abbvfi the grand mean dba'ective rating, 

■ ' ■ __<L _i. .L_ 

i High Acceptable (HA). Objectives with job analysis ratings. at the 
grand mean or within one Standard deviation above the grand mean 
objective rating. ' 



• bow Acceptable (LA). Ob jecti job analysis ratings at or 



9 



within one standard deviation below the grand mean objective 
rating; ' ._. • 
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• Least Job Related (LJRji Objectives with job analysis ratings 
less than one standard deviation below«tfie grand mean objective 
ratingi ::. ■ . \ ^ 

The purpose of the ^dob Analysfs was to determine^ the job-relatedness 
df each bbj^ Since time and essentiality were considered to be of 

.^jqiJl^l iragc^r^ance fs^cri^tfrla^-f^^ 

. a graph With rfleaH essen^^ and meart time spent 

^^^^rei^h • th^ pfptui^of the^^^^^ pd$il^orf 

of ^ach objective along both dimferisibris can be seehi ' 

Objectives which were rated. "Low Acceptable^" "High Acceptable," or 
"Preferred" on both rating scales were considered jdb-related^^^^^^ 

. . ■ . . . t ^ ' _i 

Objective Selection , the EEOG Guidelines (1978) specify that the 

content of any test 'used for persbhriel seleetibhi sUeh as a teacher 

certification test, must bear an empirical and logical relatidnship to the 

actual job content. In order to ensure that the test was dire^ 

to the content of the Business Education fields .the number of objectives - 

to be represehted on the test for each content subarea was determined to 

be proportional to the total number of job-related objectives observed in. 

■. __ ' _*_ __*_____^ ■_ . ^. ♦ .__ I 

the subarea; Specifically," a pr*bportional n |ob*reTated 

objectives (P, HA^ LA) was selected friam each subarea. '^A total of 50 

objecti ves were selected to be measured by the testi The^ selectiori of 50 



oBjecttvIs refl-ects a balance between ensuring adequate content tov^ragi 
and ensurin9( that esch objective Wii mgasufed;^ by niofs than one test • 
questibrii fhe -OU of ofe'ectives selected from, each subarea was based 
on the following fbhnu la: ' " 



Total Number of P,HAi arid UK 
gbjeetives in subarea 
Total Number of J HA, tA 
objectives in objective list 



Number of P, HA» LA objectives . 
to be selected iri stibarea 
Number of Pi HA, tA objectives 
to be selected for the test {N=50j • 



lased_gnjtM:SzJ'omul^ 
j(*as as follows:: 



Subarea 

■•■ i.i"' 
III. 

IV. 



^Number of Jp.brRel at^d- 

t4ve&- : 

27 , 

■•ST., ..V. 
23 



Number of : Selected 

^^)b4ec44^ves" ' i 

12 

■ .1-2 

■ ' 4; ; ..; . 



Develop Test Blueprint . After the number of objectives to „be 
selected from each subarea had been determined, the specific dbdeqtives 
"for which items were written were select The objectives selected for 
the test were chosen primarily based on th6 job analysis results^ In all 
cases, those objectives that were rated as preferred on both the time' 
spent and essentiality scales were selected for i tan writing. The 
remailiing objectives were selected from aro most highly rated ■ 

objectives; ^ those rated as high acceptable on of the two 

rating scales. Where two bbjectives w^^ found to rne^sur is Similar ^ 

content^ only one of the objectives was selected and another job-related 

. - _'_ / . ' _ . ■ • „ _■_ _ ' ^ j . _ . _ _ _ _• . ' ■-• 

objective was selected. to increase content coverage. " 
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The selection of all objectiyes rated as preferred on. both scales and 
the , remaining set of objectives friarit those. fated as high acceptable on one 
or ^.fii of the scales ensures that the test reflects, the most job-related' 
aspects of the field. To ensure that the number of items assigned to each 

_■ _ _'. ___ _. _ * . ' ... 

area maintains the prbpbrtib.ris of the iob -and that each objective, is 
adeqaately covered, each objective wjas assigned three items bri the test. 



In sunfiiaryi this model detenTjinfis%dse aspects of the job content 
that are job-related, selects 'a subset of' the cbritent thit is most . 
job-related to be represented on the test, ensures that the job contefit 
selected from ; each subarea bf the jbb is proportional to the amount of 
^me spent and perceived importance of th»e subarea on the job, arid 
appropriates test items to maintain these prdportidns arid to erisure 
adequate coverage of objective content. \^ : — 

\ ■ •• • 

_\__ 

The approach described provides specific guidelines to the test 



developer for developing test specifications from jot analysis results. 
The methodology presented offers a practical approach for developing 



"licensing tests specifications while satisfyirig legal arid technical 
requirements for licensing tK^^^ 



Ftiture 



There are a number of aspects of the apprdacfci described in this paper 
that kre curreritTjT^ being investigated as part of the NES Research Program 



n^this area; rndst impoHanlilyi 
tfie apprpacfi used to detennin the weighting of eaQft maqof bbritellt subareav 
included on the test. iTiese ls-sues.are 'discussed below^^ / 



' .The approach described in* this paper u^s tfle^riuiriber ;diF 
observed to be job rel ated in -eaqh subarea as a basis, for .determTning the^^_^ 
amount of weight to be givin to that subarea 'on the f inal 't.est form.v^The^ : 
number of test i terns (wei ghji rig) assf.gried^to, each ma^ or vcontent subarei .i s, 
based on tfie pirpentage of job related objective^ included .iri that . t 
siibarea. - For examtjTef if a raajbr content subarea; -contain? . . 

'objectives fdiirid to be jcib r?latld based on teacher' s'T.atingsv 20S6'bf^^^^ 
test items are assigned to tfiis subareai While the jolb aiiali&is^^^r^^ 
ultimately determine the weightTng given to each subariavt^ • 
"siz^" of the. subarea is based pri the number of object 
subarea prior to job analysis. This erisures tffat the weight- assigned to . „ 
each subarea is 'based- on sound judgments by a -panel of coritent expirts^;. • 
however,^ it provides limited input Jrito weighting f Of* the jample'of ■ , * 
iricumberits ratirig the objeetives. To ensuri greater tripu^ 'for those 



■ * • V 



r ating tBI~objectives, one cbuT^^taT n 'rati np of^^hS^^^^ bf .^eacft ^ 

itia^.or content subarea from respondents ^tn 'addition td the specific ■ ; ; . 

objective ratings obtained; Each respondent could be asked to determine; 

' -- ■ . " - ■ ■ ■ .'- , - ■ 1 '—■■^ . ■-*•■,- to ' . - - ' ' ' '* 

the importarice of edirRi subarea by assigriirig 10^ pqirits Across the 

subareas.; For examplei if,. there were four gubareas, one might, assign , the 

190 poirits as fol lOws : .subarea 1 20" points, subarea 2:' 30 points, 

subarea .3: 45 poirits, sybarea ^: 5 pOirits. „ the ayerige ra^ each 

subarea^ across responderits could Be tr^ansfonned 'back to a scale rang i 

frm l-v^lob and treated as proports that could serv^ as subarea weigh ts..^^^^ 



•15- 



Pte Ty Trig sol e^^^^^ does have drawbacks 

however i respondents may not have the broad perspective, on the field that 
•r-. is offer panel of subject area experts. Td ensure that both 

^^^^^^^: .s anea experts^ arid jo6, Iricdmberit iriput is adequately represeritedi 

:ec6nside;rat^^ could be given to -using a cfmposite rating that relys both 
. oijjTatlng obtained from suH'e^^ respondents and front the proportion of job 
■ rel^a^^ subarea^ the'NES research prograarj in 

this are c6mb\nirig tfieS^ t>(d pieces 



of -information to obtsifi subarea weights ^ v 
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